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designed to assist staff and members of the Oregon Educational 
Coordinating commission with a policy decisioti. aulti-form, 
mulf i-method and multiple audience assessment strateg'J' was used in an 
effort to account for both the technical and the political dimensions 
of analysis. The assessment featured in-mail surveys, interviews, 
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environmental factors that might affect future educational 
information and counseling service development ii^ Oregon^ a 
specification of study , limitatio'us, and a final working conference 
where preliminary a^ssessment findings were reviewed and debated. The 
needs assessment considered the,adeguacy of educational information 
and counseling services in the state- in order to help in the decision 
o^ whether tc apply for federal funds under the Educational 
Information Centers Program. It was concluded that information 
services were not adequate, ( Author /S») j 
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' \ ABSTRACT ♦ . . ' ' 

Too often policy-makers are confronted wiih results from evalMa- 
tion studies which prove of little use in policy deliben^tiiShs . The 
purpose of . this paper is .to share information about several evaluai- 
tion strategies used in a 197§ assessment study irj Oregon designed ' 
to assist staff and members of the Oregon Educati/onal Coordinating 
Commission(OE€C) with a partic^Har policy decision. 

A ratil ti- form, multi -method and multiple ai^dience assessment 
strategy was used in an efi^ort to account for both the technical and 
the /pol i ti cat dimens'ions of the study. The Assessment featured: in- 
mail surveys; interviews; extensive use of sjecondary soi#rces of data; 
/Identification of important environmental factors which might affect 

/ 

future educational information and counseling service development 

in Oregon; a specification of study limita/tions; and a final working 

, / ■ . * 

conference, where prel.i?^|i)ary assessment findings were reviewed and ^ 

debated.- . ^- / \ J 

■ " . • / - ^ ^ 

The author concludes that «vaTuati oh studies by definition rnqst 
often risk a certai n^sfegree of "(nessine^s" if eventual use \s to be 
maximized. She argues that the vision of the evaluator in an un- 
cluttered setting where everything is neatly quantifiable isWchaic, 
if it ever worked. A deliberate and careful integration Of the tech- 
nical and the political are said to ma,ike the difference between a 



useful and a relatively useless evaluation pr^uct. 
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' \ Data data data; I've got data an around. 

I've got data In ny hair. 
I've got data. on the ground. 
But I ain't got.^..I ain't got no information. 

Copyright©. ^Mary Kinnick 1980.- 
" . ' Song lyric to" Data Data Data*. 

While not grammatically correct* the sentiment expressed above cap- 
tures th^ too often heard lament of policy-makers when confrontecj* with 
results of an evaluatioli study. , Charts and Tables m^y proliferate, and 
the stu€ty Ina^y ^^ven maike for interesting reading; too often, however, it 
fails to. inform and more frequently, fv'Ts to provide information useful 
to decision-making.' 

The purpose of this' paper is to share information about several 
evaluation strategies used in a 1979 assessment study in Qre^on( Kinnick, 
1979) designed to assist staff and members of the Oregon Educational 
Coordinating CoroBi\ssion(OECC) with a particular policy decision. As is 
usually the case* time was short, s^ff and other resoiirces limited, and 
the political climate murky. Despite several important study design 
Jimitations, however,, aesults did play. a significant role in the policy 
deliberation process. The evaluation(ici this case, a needs assessment) 

design, methods uss'ed and reporting strategies should be of particular 

\ • ■ ^ - 

interest to those members of the institutional research comnunity con- 

t -I 

cerned with maximizir^ the impact of evaluation studies on decision- 
friaking. ' . 

The Situation and Study Focus , 

. ^ The Educational Information Centers Program(EIC) was authorized by 
Title. IV of the.Education Amendments ^ igyeCP.L." 94-482) . June.30, 
1978 regulations (see Federal Register, vol.43, no. 127) specified that 
these Centers' were, to make educational information, guidance, counseling. 



V 



ERIC 



and referral services accessible to all residents of the state. States 
could apply for planning and/or implementation funds(about $50,000 per^ 
state) for the first year.. 1978-79. Oregon elected to apply for and re- 
ceived dollars to- support a statewide assessment of the need for any new, 
improved or otherwise altered educational information- and counseling 
services. ' ' 

Early discussions with OECC staff members made clear the assessilient- 

f 

study Was expected to address two major questions-* Should. federal funds 
(which required a substantial state* match) be sought in the future to 
support Educational 'information Center program development *ih Oregon?, and 
If so, how should the funds be used? The study timeframe was six months. 
Staff i'ncluded one full-time director, several part-time professional con- 
sultants, and a computer programmer. \ 

Views'. held about Educa^nal Informaidon Centers were found to be 
quite mixed among various segijents of the pfctsecondary. education communi- 
ty. Some felt Oregon's ex'^effsive codimunity coll'ege system was already 
meeting all .of .the statewide ElC-related needs. Some,. feared the growth 
of yet another duplicative bureaucracy.. Others felt that tob much public 
exposure of EIC's. would only increase public demand for more services 
which, in the end, the s4:ate would be asked and refuse to pay for. Still 
others strongly felt that while state and local communities should foot 
the bill for information and counseling services for>ip-school youth, adults 
should, fend for themselves. The study began with seemingly few proponents 
of EIC in Oregon. . 

Assessment staff concluded that at minimum the study must.be designed 
to- answer the following question: What might be the consequences of doing 
nothing(e.g. not applying for future funding, which was guaranteed)? Assess 
ment staff recognized that for funding to be sought, a very strong case of 
service inadequacy would have to be made. ' 

6 ^ . 



The Criteria Challenge 

Prior to conducting the assessment, no criteria existed for use in 
both measuring and judging the adequacy of current service delivery. Federal 
EIC regulations as well as state educational "goal" documents were exai^nin- 
ed and a set of criteria developed and reviewed by OECC staff. The credi- 
ble reception and full consideration of assessment findings were amtici pa- 
ted to be highly contingent on the general acceptability of the criteria 
developed. Nine basic criteria were used to- assess current service ade- 
quacy and* are shown in Figure 1. A variety of assessment strategies^were 
then selected to collect information relevant to each of the criteria.^ 
Assessment Strategy: Integratin^Numbers and Politics / 

A multi-form, multi -method and multiple audience assessment strategy 
was.^ised in an effort to account for bpth.the technical^and the political 
dimensions of the study. ^ A paper by the League of California Cities(1975, . 

p. 13) makes the observation that: ' , . " 

^. ■ ' ♦. . 

Social ri^eds assessment is, in the final 
analysis, a political art. It requires 
political sensttivity as well as technical 
expertise. 

Figure 2 provides an overview of assessment strategy components. 

Specific inquiry methods were selected which could provide information 
about the quantity, quality and location of current educational information 
and counseling' services available to various sub-populations in the state. 
In addition to^ searching for measures and Indicators of service "adequacy" 
related to the key criteria, an effort was made to produce baseline data 
for future use in monitoring change in service delivery activitx.in the 
state. A ^rief summary of the specific assessment strategies used may 
be found in Figure 3. 

Too often, assessment studies fail to make sufficient use of data that 



Figure 1 

-Criteria Used to Assess Service 
Ade^juacy 

> 



K Educational and career information services are designed to' 
enhance -informed choice among learning and career option's. 

• . • . • I 

2. Guidance service consider client needs and interests as primary; 
and information xonrej>*-is impartial and. unbiased. 

3. Educational information services are equally available to all 
Oregonians regardless, of geographic residence, sex, ethnicity, 
socio-economic status, age, or handicap. 

4. A similar quality of educational . Information services are avail- 
able to all. current and ptospective service clients. ¥ 

5. Educational information services are designed to enhance the 

information- seeking and infqrmati on-processing skills of clientO 

I 

' 6. Printed information content is accurate and readable. 

* 7. Educational information services are delivered with a minimun of 
unnecessary duplication and a maximum of coordination and cooper- 
ation. , , . - 

8. Educational information -service users report positive inpact from 
the use of such services. * , 

.M ■ • • 

9. Educational information service providers use on-going evaluation 
processes to improve their service programs. 
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^ Figure 2 * 
Information Collection Strategies and Products 



Identify Information 
and Service -Sources 



Examine Demographic Trends, Special 
Studies and Characteristics /Needs of 
Target Populations and Regions of the 
State 



Adnl ni s to r S urvey s 
to Current and • 
Prospective Pro- 
viders of OC 



Re viiew Local and ' 
National Studies of 
Target Population 
Needs 



Interviews with 
"Panel of fexperts" 



Produce PQ^totype 
Direct(kry to EIC- 
Related Services 
in Oregon* - 

t - — r — 



Fjeld Reports from 
Regional' CA£l-affil- 
iated Staff on Local 
Populatio n Jjeeds 



Produce Analyses, Findings 



1- 



Conduct Invitational 
Conference to Review 
Findings and Consider 
Tiecommendations 



Produce Final 
Conference 
Sunmary Repo rt* 



Produce Final Report 
Including a Set of 
Reaiommendations for 
EIC-Related Service 
Development in Oregorff 



* Final Assessifent Products 
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Figure 3 
Assessflient Strategies 



Multi-Metfaeds 

1. extensive secondary literature review of previous state and 
national studies 

* 2. compilation of state demographic and social indicator data 

3. content review of printed information ibout educational /train- 
ing (5pportunities.» financial ai^ and coHs. etc. made avail- 
able to prospective students by postsecondzyry education insti- 
tutiorrs , 

# ♦ 

4. design and administration of in-mail surveys for use by Service 
providers (postsecondary education institutions* high'school" 
counselors, school and public librarians, social service and 

^ community-based organizatiorjs/agencies, and employers) ' 

5. compilation of a prototype educational information, and counsel- 
ing services directory showing, by county, the kind and extent 

-of services offered by different agencies * 

^ 6. individual and small group interviews with key service providers 
and service program administrators 

7. 'identificajtion of; major environmental factors which could affect 
EIC- related future developments 

Multi- Forms 

use of five different in-mail surveys which included , both 
common and unique items 

«■ • * 

* Multiple Audiences ' " 

1. direct EIC- related service providers 

2; service staff who work with particular sub-populations but 
who currently offer little ElC-related service 

3. service program administrators' 

4. individual clients via review of numerous other studies of 
consumer needs 

5. advocates for particular consumer groups / \ 



alreaciy lexis ts as well as findings from previou's studies. Instead, the 
piore conroon' approach is to de^gn yet anbther survey 'lair* interview schedule 
(acting as if others did not'exist), sampile some^ populations, and admin- 
ister the instrtiraent. Wheels are reinvented everyday, /tx^ensive use was ' 
made^in this assessment of secondary data source's. Jhe ^uKeys, iiit^rviews 
and content review strategies Were designed to collect new information* 
The jaterviews provided more infS.rmatiQri about and enhanced sensitivity 
,to sources of disagreement among agency ref^refbntatives abou* the adequacy 
of current services. Sensitivity to t/ie political climate *fas essential . 
when final recommendations were prepared. 

The failure^to account for, in. a direct way, the current satisfaction 
with services among Oregonian consumers proved to be somewhat of a liabili- 
ty. Despite the fact that a large number 6f othe»* states had surveyed 
consumers about their educational/career information and counsel in'g needs 
and reported very similar findings(see Cros-s in Peterson, 1979), some 
assessment stuc/y reviewers persisted" in arguing that the situation might 
well be different in Oregon. (As a now adopted 'Oregonian, the authir must 
confess to a growing conviction that indeed we Oregonians are" different! ) 
The lack of Oregon-specific data about consumer satisfaction with curreryl 
services was felt by a few reviewers to weaken overall stbciy impadt. 

While lack of this information did weaken tfi^ credibility of some - ' 
study findings, overaU the technical rigor of the study remained intact. 
As institutional researchers know, however, technical rigor and use of 
commendable research^^cedures do not automatically resultj^in a product 
which has impact . It is critical to remain sensitive to the fact that the • 
sources from whichyinformation is collected are themselves differentially 
valued. / - - 

In di^afting the firi'al report section on assessment objectives and 



methods; considerable attention was giv^n to detalHIig study limitations. 
OECC staff reviewers stated this particular section was w^M received and ^ 
is uslially not 'included/in such reports; By highlighting the limitations, 
various potential sources -of criticism of the study were anticipated and. 
to a great extent, diffused. A statement of limitations also allowed the 
reader to know what degree of confidence could be placed in the- various • 
findings and .to review a set of issues for which further ^sessment studies 
were needed. Little could be said, for instance, about patterns of post- 
secondary education participation by different age and racial groups due 
to inconsistent, inadequate and inaccurate institutional , state and federal 
-data collection and reporting practices. a . 

Finally, a special chap^r in the final report was devoted e'xclusivei^ - 
5^to a review of environmental factors which were identified and then considered 
during the review of service adequacy and in the formulation-of recommenda- 
tions for further ElC-related development in Oregon. The viability of final 
recommendations was considerably enhanced by an accounting of the various 
economic, social and political factors listed in Figure 4. * 
Analysis: A Legal Brief Approach ^ ^ 

The multiple/Sources of Information collected were reviewed in light 
of the "service adequacy" criteria outlined earlier in this paper. The 
analysis process was both highly systematic ^nd "gestalf-1 ik^. Infomiation 
from all data sources, the qualitative and quantitative, was reviewed ^for 
coQsistencies and ^discreMncies across the study methods. 

Various conclusions which seemed suggested from an initial review ofv 
the findings were formulated. The data was then re-examined systematically 
for evidence which efther supported or did not support the initial conclu- 
sions. Som initial conclusions were eventually discarded and others alter- 
ed to better'fit with the nature and extent of evidence which emerged. Par- 
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^ Figure 4 " 
./Environmental ^Factors and EIC--R6lated Impacts 



factors - 



/ ■ 



EIC-Related Impacts. 



energy costs, lower availability 



» 2. ra'pl developments in the 
telecommfinti Nations field 



3. increased competition for students 
among institutions 



4. break^wij of lock-step of school, 
work, leasure 



5., continued population growth in 
' Oregon; IncreHse, through, in- 
mig ration, of both those with 
a high level of formal educa- 
tion and those without . 



6. in the future, expectation 
that there will be fewer jobs 
with career'vpotentfal ; rapid 
increase in Jjob-market entry 
by women 

7. increase in inflation rates, 
impending recession 



8. ballot box rejection of more 
government spending 



9. public support /or decentrali- 
zation of program and service 
administration*, ' ^ 



access to. EIC must be less ' 
dependent on the car; explore 
home delivery sites and oth,er 
sites close to home ov work 

• potential of increased af ford- 
ability of home and/or work- ' 
.site based service access • 

A* ■ - m 

potential, for deteriaration In 
qualify of information made 
available by ih'sti^utions to 
'prospective students; more in- 
centive to '**ien" and less j;c^ 
"inform"' 

increase service access so 
individual's can make -use of 
them when* they feel ^the need 

jncreasing^emand as well as 
need ^f or education/ training 

'■ ' ' 

many newcomers Tess familiar 
with In^place service networks; 
need' for more otjtreach 

increase in interest In moving 
laterally in the job.stru9ture; 
greater Interest in transfer- 
ability of current skills^ 



less value plated on education 
relat1\^e to other goods and 
services unless direct economic 
pay-off for more schooling shown 

^l<f ttle.#hope for new dollars., to 
support EIC; support must cpme 
from current resources 

increase In demand for local 
control of service developn^nt 
ant* operation 
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tlclpapt^ at a concluding working conference also reviewed preliminary findings 
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^ t and conT:lusioj>s land offered extensive comment. "Their input was reviewed and - ^ 

-played a role th the revision of seme conclusions. 

^ This 'approach to the analysis, al^l owed for a review of-$flf^ multiple 
^ sources of inforrtiarron jis^d^ih the study with a focus on a partictjlar con- 

ter)tion(e.g. a :stateraent of conclusion) about the nature of current EIC- < \ 
V* related ^vice in Oregqn; An^altemate app^aeach often used in these kinds 
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of studies, is to review separately a nyriad of findings fran each of the 
different assessment methods used(e.g. survey results; interview results; etc. ). 
The result l^s a highly fragmented report oj findings which. may or. may not 
seem to relate directly to the fundamental issue(s) the stu(^y was to Address. 
The "legal-brief analysis and reporting approach used in this assessment ' 
required an integration <0f findings . Fourteen major conclusions were pre- " 
seated, along with supportive evide'nce and arguments. - 
Developing Final Recommendations:" A Pol iticaV and Technical Process 

* The stu<|y .concluded that indeed current EI C- related services were not 
adequate and destribed the ways in which current services were inadequate. 
It further' argued that there was strong evidence to suggest that doing nothing 
to imprsye services iiras likely to contribute to an increase in^rrent social 
and economic inequities among var^ious sub-groups i^the populations 

Two special strategies were used to develop k'nal recommendations to 
the OECC. First, assessment study staff reviewed th^ upcoming activity 
plans of several major service providers' already operating in the state, 
including the O*^gon Occupational Information Coordinating Committee, the 
State Library, the Oregon Department of Education, the Oregon Stat^ System 
of Higher Education, Statewide CET/^ and several primes, the State Scholar-" 
shin Commission, ^d the Consortium for the Advancement of Experiential and 
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Lifelong Learning, . - 

♦ 

Second, a working conference was conducted to which representatives 
of the OECC staff, major service providers<Bnd consumer advocates were 
ImitedC Franklin. 1979). Thirty-seven persons attended the two-day con- 
ference which was organized to achieve the following: to provide an oppor- 
tunity for debate oh the releveince and significance 9f major assessment 
findings; to corroborate stuc^y staffs* conclusion that there were con- 

A, 

siderable gaps In current service adequacy; and to provide a forw fOr 
additional reflection on whlTt should be , recommended for the future. The 
extent of concurrence on each of the preliminary findings and conclusions 

of the study was solicited and later reviewed as the final report was 

f - ■ ■ ■ ■ ■ 

prepared. Participants discussed and debated a series of alternative EIC 

* 

program plans and structures giv6n,an assumption of a known amount of 
fe^deral dollar availability. By-products of the conference were the identi- 
fi cation of ways in which current service providers were willing to conriit 
their agencies' to cooperative EIC- related activity in the future and the 
Increase In awareness among those attending of what other agencies do. 

The working conference not only resul ted in a technically improved 
final assessment report, but greatV enhanced its credibility. Also, ;the 
alternative recommendations proposed in the final report were not "pie in 

* 

the sky" ideas but reflected both the assessment findings and current 
political and economic realities in the state. ~~ 
Final Reporting - . 

An Executive Summary of ' the assessment findings was considered an 
essential supplement to the final report. A detailed slimmary of the inter- 
actions during and the results from the working conference was also. pre- 
pared. After the OECC staff and members had reviewed the assessment docu- 
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ioents, assessment staf^ gave a suirmarybral presentat;jon of the findings to 
'the full Commissrfon. A special oral presentation of the findings, however, 
was not given to OECC staff members but siiould have been. Severaf* problem- 
:atic. issues arose after release of the study which could have been easily 
cleared upt had an oral presentation taken place, ^ ' 

- " * 

A Final Conroent • . > 

Evaluation studies oftentimes by definition mbs^t risk some degree of 
"liTessiness" if 4:heir eventual use is'to be maxipiized. The vision of the 
evaluator locked away in a laboratory, reviewing computer printouts, 
constructing Tables^ ^nd keeping everything neat and tidy is sinply archaic, 
(if it, ever "worked). A deTiberate and careful integration of tlie technical 

r • . • • 

* ■ ■ fc . . ■ 

and the pol i tical (wi th some artistry tftrown in for good tneasure) in the 
construction of evaluation^-de^gns can mean the difference between products 

Ge. Most of us would prefer to make 

V 



which make or do not make a di 
'• ■ • . ^ 

-a difference. - 
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